Skip to content

Add ParquetExporter (Vortex -> Parquet) - #362

Merged
dfa1 merged 1 commit into
mainfrom
feat/parquet-exporter
Sep 3, 2026
Merged

Add ParquetExporter (Vortex -> Parquet)#362
dfa1 merged 1 commit into
mainfrom
feat/parquet-exporter

Conversation

@dfa1

@dfa1 dfa1 commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Summary

  • New ParquetExporter/ExportOptions in the parquet module — the inverse of ParquetImporter — writing a Vortex file to Parquet via Hardwood 1.1's new ColumnWriter/ParquetFileWriter write API.
  • Flat schemas only for v1: Bool, non-F16 Primitive, Utf8, Binary, and vortex.timestamp (MILLIS/MICROS/NANOS). Struct/List/Map top-level columns throw UnsupportedOperationException — nested export is a documented follow-up.
  • Bumps dev.hardwood:hardwood-core 1.0.0.Final → 1.1.0.Beta1 (adds the write API this depends on, plus read-path speedups for ParquetImporter).
  • Docs updated in the same commit: docs/reference.md, docs/how-to.md, CHANGELOG.md.

Test plan

  • ./mvnw verify -pl parquet -am — unit tests (29 in ParquetExporterTest: type mapping, flat export + projection + unsupported-type rejection, and two Vortex→Parquet→Vortex round-trip tests via ParquetImporter, covering U8/U32 boundary values and the timestamp extension)
  • ./mvnw verify — full reactor build, all modules green
  • ./mvnw verify -pl integration -am -Dit.test=DocsConsistencyTest — living-docs claims still resolve

🤖 Generated with Claude Code

https://claude.ai/code/session_01LjU7zWghUiJauRxK58nyUs

@dfa1
dfa1 force-pushed the feat/parquet-exporter branch 3 times, most recently from 4241c98 to f3bd665 Compare September 3, 2026 04:39
Inverse of ParquetImporter, built on Hardwood 1.1's new ColumnWriter/
ParquetFileWriter write API. Flat schemas only: Bool, non-F16
Primitive, Utf8, Binary, and vortex.timestamp (MILLIS/MICROS/NANOS);
Struct/List/Map top-level columns throw UnsupportedOperationException.

- Bumps dev.hardwood:hardwood-core 1.0.0.Final -> 1.1.0.Beta1 (adds the
  write API this depends on, plus read-path speedups for ParquetImporter).
- ParquetExporter / ExportOptions in the parquet module, mirroring
  ParquetImporter/ImportOptions' shape.
- vortex-reader promoted from test- to production-scope dependency in
  parquet/pom.xml.
- ParquetExporterTest: type-mapping unit tests, flat export + projection
  + unsupported-type integration tests, and two round-trip tests
  (Vortex -> Parquet -> Vortex via ParquetImporter) covering flat types
  (including U8/U32 boundary values) and the timestamp extension.
- CLI: the `export` subcommand now dispatches to ParquetExporter when the
  output path ends `.parquet` (previously CSV-only); usage text and
  ExportCommandTest updated.
- Remote HTTP(S) support, both directions:
  - ParquetExporter.exportParquet(VortexHandle, Path[, ExportOptions]):
    exports an already-open handle (local VortexReader or remote
    VortexHttpReader) without an intervening local copy; the handle is
    not closed here, the caller keeps ownership.
  - ParquetImporter.importParquet(URI, Path[, ImportOptions]): imports a
    Parquet file served over HTTP(S), fetched entirely through targeted
    Range requests via the new HttpInputFile (implements Hardwood's
    InputFile), mirroring VortexHttpReader's own range-fetch pattern. No
    full-file download occurs.
  - checkNoDuplicateNames/fillRow take a String sourceName instead of a
    Path, since a remote source has no filesystem path to report in
    error messages.
  - CLI: `export`/`import` accept a url source too (Parquet-only in both
    directions; CSV and stdout streaming from a URL are out of scope).
  - HttpInputFileTest covers the HEAD/Range request mechanics against a
    mocked HttpClient. Live-verified end to end against a real ~3M-row
    public Parquet file (NYC taxi trip data) via the built CLI jar:
    remote Parquet -> local Vortex -> local Parquet -> reimport, with
    row counts and min/max stats matching exactly at every step.
- docs/reference.md, docs/how-to.md: document the new class/options, the
  remote overloads, and the CLI's Parquet import/export support.
- DocsConsistencyTest: allowlist WriterConfig as a known external
  receiver cited in reference.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LjU7zWghUiJauRxK58nyUs
@dfa1
dfa1 force-pushed the feat/parquet-exporter branch from f3bd665 to 3d38a8d Compare September 3, 2026 04:55
@dfa1
dfa1 merged commit 939084b into main Sep 3, 2026
6 checks passed
@dfa1
dfa1 deleted the feat/parquet-exporter branch September 3, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant